Use of Affective Visual Information for Summarization of Human-Centric Videos

The increasing volume of user-generated human-centric video content and its applications, such as retrieval browsing, require compact representations addressed by the summarization literature. Current supervised studies formulate a sequence-to-sequence learning problem, existing solutions often neglect surge view, which inherently contains affective content. In this study, we investigate affective-information enriched task for videos. First, train visual input-driven state-of-the-art continuous emotion recognition model (CER-NET) on RECOLA dataset to estimate activation valence attributes. Then, integrate estimated emotional attributes their high-level embeddings from CER-NET with information define proposed (AVSUM) architectures. addition, use attention improve AVSUM architectures propose two new based temporal (TA-AVSUM) spatial (SA-AVSUM). We conduct experiments TvSum COGNIMUSE datasets. attention-based TA-AVSUM architecture attains competitive performances strong improvements videos compared in terms F-score, self-defined face recall, rank correlation metrics.